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METHOD AND APPARATUS FOR PROVIDING VIRTUAL PROCESSING 
EFFECTS FOR WIDE-ANGLE VIDEO IMAGES 

This application makes reference to, incorporates herein and claims all benefits 
accruing under 35 U.S.C. § 1 1 9(e) by virtue of a provisional patent application earlier 
filed in the United States Patent and Trademark Office on April 8, 1999, entitled System 
and Method For the Capture and Display of Spherical Video Images, (Attorney Docket 
No. 1096.79941). This application also incorporates by reference the following U.S. Pat. 

Applications filed on the same date as the present application: Serial No. , (Attorney 

Docket No. 01096.86946) entitled "Remote Platform for Camera"; Serial No. 

(Attorney Docket No. 01096.86942) entitled "Virtual Theater"; and Serial No. , 

(Attorney Docket No. 01096.84594) entitled "Immersive Video Presentations". Also, 
U.S. Pat Nos. 5,990,94 1 and 6,002,430 are incorporated by reference. 

TECHNICAL FIELD OF THE INVENTION 
The present invention generally relates to processing effects for wide-angle 
images. More particularly, the present invention relates to special effects processing that 
provides image portion replacement for wide-angle video images, shifted image portion 
replacement for wide-angle video images and other special effects for wide-angle video. 

BACKGROUND OF THE INVENTION 
A goal of imaging technology is to make the observer feel as though he or she is 
part of an image. U.S Pat Nos. 5,990,941 and U.S. Pat. No. 6,002,430, currently assigned 
to the assignee of the present application, describe wide-angle image viewing systems 
and methods. The patents provide full freedom viewing inside of a complete spherical 
image composed of one or two combined hemispheric images. As a result, there are no 
bounds on the user's freedom to view in any direction. 

U.S. Pat. No. 6,002,430 patent discusses techniques for capturing first and second 
images, each having a field-of-view of 180 degrees or greater. In one embodiment two 
cameras are placed in a "back-to-back" arrangement having lenses with a 180 degree or 
greater field of view (e.g., fisheye lenses). An imaging element or elements capture the 
images from each lens. The captured images are seamed together to form a single 
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spherical image. Also, U.S. Pat. No. 6,002,430 patent includes a viewing engine that 
provides a real-time interactive image window that can be directed anywhere in the 
spherical image. Preferably, a personal computer system runs perspective correction 
algorithms to create the image window. Other wide-angle viewing systems are known, 
but known systems generally do not provide special effects for wide-angle video images. 

Reber et al., in U.S. Pat. No. 5,706,421, discloses placing an animated character in 
a wide-angle image. An animated character generator superimposes the animated 
character onto a perspective corrected image in a spherical wide-angle image. However, 
the animated character is over-laid on the spherical image, and the Reber et al. does not 
disclose performing special effects on non-perspective corrected images in wide-angle 
video images. 

Special effects can be used to enhance the quality of video images and a need 
exists for providing special effects for wide-angle video images. 

SUMMARY OF THE INVENTION 
An aspect of the present invention is to provide an apparatus and methods that 
provides special effects for wide-angle video images. 

In accordance with the principles of the present invention, a method is provided 
for capturing wide-angle video that includes a plurality of sequential frames, each frame 
having at least one wide-angle image, and replacing a portion of at least one of each 
wide-angle image in at least one of the frames with a replacement image. The step of 
replacing a portion of at least one of each wide-angle image includes replacing the 
portion with a portion of a wide-angle image in a second frame captured subsequently to 
the frame having the replaced portion. The portion of the image in the second frame may 
have the same shape as the replaced portion and represents an unobstructed view of the 
surrounding environment. Alternatively, the step of replacing a portion of at least one of 
each wide-angle image includes replacing the portion with a planar image that is waiped 
to the same curvature of the replaced portion and has the same shape as the replaced 
portion. 

In accordance with the principles of the present invention, an apparatus is 
provided that includes a wide-angle video image capture device that captures wide-angle 
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video in a plurality of sequential frames, each frame having at least one wide-angle 
image, and preprocessing circuitry that receives digital wide-angle video images of the at 
least one wide-angle image in each frame and replaces a selected portion of at least one 
of each wide-angle image in at least one of said frames with a replacement image. The 
5 replacement image includes a selected portion of a wide-angle image in a second frame 
captured subsequently from the frame having a replaced portion. The selected portion of 
the replacement image has substantially the same shape as the replaced portion and 
represents an unobstructed view of the surrounding environment. Alternatively, the 
replacement image is a planar image warped to the same curvature and having the same 
10 shape as the replaced portion. 

In accordance with the principles of the present invention, a method and apparatus 
is provided for performing image-intensity scaling. A wide-angle video image is divided 
into a plurality of areas. An adaptive threshold, associated with the average image 
intensity of pixels in each area, is calculated for each area. The intensity of a pixel in at 
15 least one of said areas is compared to the adaptive threshold for the area. The pixel 
intensity is adjusted, when the intensity is not equal to the adaptive threshold. 

In accordance with the principles of the present invention, a method and apparatus 
is provided for detecting and removing flare. Preprocessing circuitry compares the 
change in intensity of pixels in each row of at least one wide-angle image in each frame 
20 to two threshold values to detect flare, and adds pixels in each frame to remove the 
detected flare. 

BRIEF DESCRIPTION OF THE DRAWINGS 
The invention is described in detail with regard to the following figures, 
25 wherein like numeral references refer to like elements, and wherein: 

FIG. 1 is a schematic block diagram of a wide-angle video image system in 
accordance with embodiments of the present invention; 

FIG. 2 is a schematic block diagram of an image capturing system in accordance 
with embodiments of the present invention; 
30 FIG. 3 represents two spherical images combined into a single spherical image in 

accordance with embodiments of the present invention; 
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FIG. 4 represents a data structure of a video frame in accordance with embodiments 
of the present invention; 

FIG. 5 represents an image portion replacement special effect in accordance with 
embodiments of the present invention; 

FIG. 6 represents a shifted image portion replacement special effect in accordance 
with embodiments of the present invention; and 

FIG. 7 shows flare in a wide-angle video image in accordance with embodiments of 
the present invention. 

FIG. 8 illustrates a sampling and blending operation to fill a blank section of an 
image in accordance with embodiments of the present invention. 

FIG. 9 illustrates a helicopter with a camera system mounted thereon according to 
embodiments of the present invention. 

FIG. 10 illustrates creating a 3-dimensional area in which a helicopter can be 
controlled according to embodiments of the invention. 

FIG. 1 1 illustrates a seaming operation according to embodiments of the invention. 

DETAILED DESCRIPTION OF THE INVENTION 
A system for creating wide-angle video images, such as spherical video images is 
shown in FIG.l. Wide-angle video image system 100 includes video image capture 
system 1 10, preprocessing circuitry 120, viewing engine 130 connected to display 
monitor 140 and/or storage 150 connected to preprocessing section 120. Video images 
output from image capture system 1 10 are processed, possibly along with other data, in 
preprocessing circuitry 120. Preprocessing circuitry 120, for example, adds special 
effects to the received video images. The output of pre-processing circuitry can be stored 
in storage 150 and/or processed in viewing engine 130, such as the viewing engine 
disclosed in US. Pat. 6,002,430, for display. 

Image capture system 1 10 captures at least one wide-angle image. For example, 
any of the video image capture systems described in co-pending U.S. PaL application 
(Attorney Docket No. 1096.82594) may be used. The output of image capture system 
110 may be frames of film, each frame including at least one wide-angle image. The 
video images are either captured with a digital image capture device or converted to 
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digital images. For example, as described in co-pending U.S. Patent Application Serial 

No._ (Attorney Docket No. 1096.82594) and as illustrated in FIG. 2, an image 

capture device in image capturing system 1 10 may include two video cameras 210 and 
220 having fisheye lenses 250 and placed in a back-to-back arrangement for 
5 simultaneously capturing two video images. Video cameras 210 and 220 can be 

mounted, for example, on a base 230 attached to robot platform 235 located on mobile 
robot 240. Other structures for supporting video image capture devices, such as a tripod, 
may be used, and other vehicles for carrying an image capture device, such as a 
helicopter (as described in U.S. Patent Application Serial No. . (01096.86946) 

10 entitled "Remote Platform for Camera" filed herewith), may be used. Video camera 210 
is orientated to capture the environment towards the front of robot 240 and video camera 
220 is oriented to capture the environment towards the rear of the robot 240. Video 
cameras, however, can be orientated in other opposing directions. For example, co- 
pending U.S. Patent Application Serial No. (Attorney Docket No. 1096.86946) 

15 describes a platform on a robot for carrying an image capture device. Video cameras 210 
and 220 capture the environment digitally or on film in two hemispheric images 310 and 
320, which are eventually seamed together to form a spherical image 330 shown in FIG. 
3B. According to one embodiment, as illustrated in Figure 1 1 , the raw video images of 
the first two hemispherical images 1 100 can be corrected in 1 104 and then seamed 

20 together using the correction determined in 1 1 04 to form seamed images 1 1 02. The next 
N frames can then be corrected using the correction determined for the first frame in 
1 106. Thus, the correction for every frame need not be determined. Rather, the 
correction factor need only be determined every 10, 20, 50, etc frames, as conditions 
dictate. 

25 Image capture system 1 1 0 preferably records data representing pan, tilt and rotate 

coordinates of the image capture device simultaneously with the captured images. 
Similarly, time of day and date clock data may preferably be recorded simultaneously 
with captured images. Also, audio data may be recorded simultaneously with captured 
images. Preferably, a digital or electric microphone may record audio data and be 

30 mounted and directed in the same direction as a camera lens to capture the sounds 
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coming from the direction of image capture as known in the motion picture and television 
programming arts. 

As described above, the camera system may be mounted on a vehicle such as a 
helicopter, as illustrated in Figure 9. The camera system 901 is mounted on a camera 
mount 903 which is connected to the helicopter 905. The video and audio signals 
captured by the camera system 901 are recorded in a memory 909. The recorded signals 
can then be downloaded once the helicopter returns from the trip. Alternatively, some or 
all of the data can be transmitted back to a home base from the helicopter 905 using.for 
example a transceiver 911. The recorded signals may be transferred to the transceiver 
91 1 by a computer 909 connected to both the memory 907 and the transceiver 911. It 
will be understood that the memory 907 may be part of the computer 909. The recorded 
signals can be transmitted in either a corrected format or an uncorrected format, wherein 
the computer 909 performs the correction. In addition, the entire image signals can be 
transmitted using high resolution or low resolution. In addition, only part of the image 
signals need be transmitted back to the home base, for ©cample, only the images from the 
forward looking camera or a portion of the images in the vicinity of the direction of view. 
One advantage of only transmitting a portion of the entire images is that the decrease in 
transmission time allows the vehicle to be remotely piloted over long distances. 

Likewise, global positioning system data may be captured simultaneously with 
images or by pre-recording or post-recording the location data as is known from the 
surveying art. The precise latitude and longitude global coordinates of each image as it is 
captured can be recorded. 

The global positioning system data may be used to create a 3 -dimensional area in 
which a vehicle, such as a helicopter, can operate. For example, a cruise ship 1001 may 
offer passengers the ability to take a "virtual" helicopter ride to a nearby island 1007 as 
illustrated in Figure 10. By using the global positioning system data, a box 1005 within 
which a helicopter 1 003 can fly can be created. The passenger can then remotely pilot 
the helicopter within the defined box. The system can be defined such that if the 
helicopter leaves the box 1005, the helicopter will immediately return back into the box 
or return to the cruise ship. 
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Preprocessing circuitry 120 performs special effects, described in the following 
sections, on the received wide-angle images. Also, preprocessing circuitry 120 includes a 
processor and memory that may be provided at the video capture system 1 10 or remotely 
at, for example, a personal computer. Preprocessing circuitry 120 can assemble received 
5 wide-angle video images and other recorded data, for example, in video frames having a 

data structure described in co-pending U.S. Patent Application Serial No. 

(Attorney Docket No. 01096.84594) entitled "Immersive Video Presentations" and 
shown in FIG. 4. Also, preprocessing circuitry 120 may convert wide-angle video from 
image capture device 110 into conventional formats, such as AVI or MPEG. 

10 The wide-angle video images, such as the hemispheric images, and.associated 

data may be stored in storage section 150. Storage section 150 may include video tape, 
CD-ROM, semiconductor devices, magnetic or magneto-optical disks, or laser disks as 
the storage medium. Also, the wide-angle images may be displayed using viewing 
engine 130, which may include the viewing engine in U.S. Pat. No. 6,002,430 patent. 

15 As described above, preprocessing circuitry 120 can provide special effects for 

wide-angle images captured by video image capture system 110. In a preferred 
embodiment of the present invention, an image portion replacement special effect for 
wide-angle images is provided using preprocessing circuitry 120. 

Image portion replacement includes selecting at least one warped wide-angle 

20 image from image capture system 110 and selecting a portion of the wide-angle image for 
replacement with a predetermined image. For example, using the arrangement described 
above and shown in FIG. 2, video cameras 210 and 220 having fisheye lenses capture 
wide-angle video images that are warped (e.g., the captured wide-angle video images are 
substantially hemispheric), as shown in FIG. 3 A. FIG. 5 illustrates image portion 

25 replacement for wide-angle image 320 from video camera 220. The video images are 
mapped using, for example, spherical coordinates, and a portion 510 of the images is 
selected for replacement using, for example, the spherical coordinates. Selected portion 
510 is removed and a predetermined image 520 is placed in the same location as selected 
portion 510. Predetermined image 520 can have the same shape as the selected portion 

30 510 and is warped to have the same curvature of the selected portion of the wide-angle 
image. 
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The waiped predetermined image 520 that replaces selected portion 510 of the 
wide-angle image may be created, for example, using a conventional computer-aided 
design (CAD) program having the capability to create warped images. Alternatively, a 
planar image may be created, for example, using a well-known paint program and waiped 
to have the same curvature of the wide-angle image. For example, transforms, which 
maybe derived from transforms disclosed in U.S. Pat. No. 6,002,430 patent, can be used 
to convert the planar image to a waiped image that has the same curvature as the selected 
portion of the wide-angle image. Also, the planar image that is warped can have the 
same size as the selected portion for replacement. 

Additionally, the image portion replacement special effect may be provided for 
video images and still images. For video images, a selected portion in one or more wide- 
angle images in one or more frames of the video images is replaced with warped 
predetermined image 520. 

In another preferred embodiment of the present invention, shifted image portion 
replacement for wide-angle video images is provided using preprocessing circuitry 120. 
Shifted image portion replacement, similar to the image portion replacement special 
effect described above, allows a selected portion of a video image to be replaced with a 
selected portion of another captured video image. For example, when wide-angle video 
images from image capture system 1 10 are stored and/or converted to a video file, which 
may include conventional formats such as AVI or MPEG, a selected portion of a wide- 
angle video image from one video frame can be replaced with an associated selected 
portion of a wide-angle video image from another video frame. 

FIGS. 6A-6B illustrates shifted image portion replacement. Using, for example, 
video image capture system 110 illustrated in FIG. 2, video cameras 210 and 220 capture 
the environment in the forward and aft directions of robot 240 as robot 240 moves 
through the environment. The captured video images include a series of frames that 
include frames 610-630. Each frame includes, for example, two hemispheric images 310 
and 320 captured from video cameras 310 and 320 respectively. Frames 610-630 can 
include other information (not shown) associated with each hemispheric image set, such 
as position and audio data, as described above and illustrated in FIG. 4. 



8 



WO 00/60853 PCT/US00/09464 

As described above, hemispheric images 310 and 320 represent wide- 
angle video images having a field-of-view of at least 180 degrees. Therefore, a portion of 
the video images may be obstructed, for example, by a structure supporting the video 
capture device or a vehicle carrying the image capture device. For example, video 
5 images from video cameras 2 1 0 and 220 are obstructed by platform 235 of robot 240. 
Frames 610-630 each include front and back hemispheric images 310 and 320 
representing the surrounding environment. Platform 235, captured at the bottom of each 
image, obstructs the field-of-view for the environment below platform 235. However, 
objects in the environment obstructed by platform 235 in images 622 and 624 were not 
10 obstructed in image 612 in the previous frame. The obstructed areas of images 622 and 
624 are 626 and 629, respectively. The images being blocked by 626 and 629 were 
previously recorded in image 612 as images 617 and 618. In order to approximate the 
blocked images in images 622 and 624, the system replaces image area 626 with image 
area 617 and replaces image are 629 with a rotated version of image area 618. Likewise, 
15 the blocked areas of the image in images 626 and 629 may be obtained from image 622. 
Another example is illustrated in Figure 6B in which the camera system is 
mounted, for example, below and in front of a helicopter, wherein the forward image is 
not blocked and the backward image is partially blocked along the top of the image by 
the helicopter and/or camera mount In this example, a forward image is used to correct a 
20 fill in a blocked region of a backward image of a previous frame. For example, in the 
present frame j, the forward image 660 is not blocked by the helicopter but the backward 
image 662 is partially blocked 664 by the helicopter and/or mount. The image area 668, 
which is an inverse of the blocked area 654 in image 652, may be obtained from image 
660 and rotated by 180 degrees to approximately fill the blocked image 654. Likewise, 
25 the image area 676 may be used to approximately fill the blocked image area 664 in 

image 662. It is appreciated that the forward looking image may need to be converted in 
dimension and shape to comport with the outline of the supporting structure. 

According to another embodiment of the invention, the portions of the images that 
are blocked by a camera mount or a transportation vehicle can also be filled by sampling 
30 the regions around the blocked image and creating a blending image as illustrated in 
Figures 8A-8B. In Figure 8A, the portion of the image that is blocked is represented by 
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810. A region around the blocked image can then be designated by the system as the area 
820. As illustrated in Figure 8B, samples 830, 840 and 850 can be taken from the area 
820 and blended together to fill in the region 810. An illustrative sampling and blending 
process is disclosed in PCT Application No. PCT/US99/07667 filed on April 8, 1999, 

5 which is incorporated herein by reference. 

A subsequent frame. that includes objects of the environment previously 
obstructed by, for example, a video image capturing device support structure, can be 
identified using the velocity of the vehicle carrying the image capture device, the shape 
of the selected portion of the video image representing the obstructed objects and the 

10 image capture rate of the video image capturing device. In addition the position of the 
unobstructed objects in the subsequent frame can be identified using the same 
information if the vehicle travels in one direction between the time the video image in the 
frame having the obstructed objects is captured (e.g., frame 610) and the time the video 
image in the subsequent frame having the unobstructed objects (e.g., frame 630). 

15 Therefore, it may be necessary to increase the image capture rate when the vehicle 

carrying the video image capture device is travelling at a high rate of velocity in order to 
minimize nonlinear movement of the vehicle between the time the video images in the 
relevant frames are captured. Alternatively, if position data for the vehicle carrying the 
video image capture device is available, such as provided by a conventional GPS 

20 receiver, the position of the unobstructed objects in a subsequent frame may be 

determined regardless whether the vehicle changes direction between the time the video 
images in the relevant frames are captured. 

Also, preprocessing circuitry 120 can compare the image intensity of the 
replacement selected portion, such as selected portion 650 in frame 630, with the image 

25 intensity of surrounding portions of the video image, such as portions of video image 
310 surrounding selected portion 650 in frame 610. If the comparison yields a contrast 
larger than a predetermined value, a predetermined image, such as predetermined image 
520 described in the previous embodiment, can be used as a replacement portion. The 
predetermined image can have the same shape and is warped to the same curvature of the 

30 selected portion. Alternatively, if the comparison yields a contrast larger than a 

predetermined value, the selected portion may not be replaced. Also, instead of image 
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intensity, other aspects of the selected portion or the entire image that are known in the 
art (e.g., aspects related to image quality) may be evaluated for determining whether to 
replace the selected portion of the video image. 

Additionally, other special effects can be performed using preprocessing circuitry 
5 120- For example, preprocessing circuitry 120 can provide image intensity scaling for 
one or more wide-angle images in each video frame that emulates the functionality of a 
mechanical iris by shifting the intensity of the displayed video image based upon, for 
example, predetermined parameters or commands entered by the user. Image intensity 
may vary throughout a wide-angle vide image. For example, the image may be bright on 

10 top and dark on the bottom or bright in the center and dark on the edges. Preprocessing 
circuitry 120 can change the image intensity of pixels to blend the bright and dark areas 
of a wide-angle video image. The wide-angle image can be divided into a plurality of 
areas. The average image intensity of each area is calculated. The image intensity of 
each pixel in each area is compared to, for example, the average image intensity of each 

15 pixePs respective area. The image intensity of each pixel may be adjusted accordingly 
(e.g., adjusted to be brighter or darker), if the pixel's image intensity is not equal to the 
threshold. The threshold can, for example, be a range of values associated with the 
average intensity of the respective area. 

Preprocessing circuitry 120 may also provide flare detection and removal. FIG. 7 

20 shows flare 710, comprised of a vertical line of high intensity pixels, for wide-angle 
image 720. Vertical edge detection can be applied for each wide-angle video image in 
each frame for detecting flare. Each image includes a plurality of rows of pixels. The 
difference, for example, between the average intensity of each pixel in a row is compared 
to a first threshold that, for example, allows for a relatively large change in intensity to 

25 detect high intensity pixels. If an entire vertical line of pixels having abnormally high 
change in intensity is not detected, another comparison can be made to a second, less 
sensitive threshold, that allows for a relatively smaller change in intensity to detect pixels 
having abnormal high change in intensities. The results of the two comparisons can be 
correlated to detect flare 710. Once flare is detected, pixels having an intensity relative 

30 to the intensity of pixels proximate the flare may be filled in each row across the flare. 
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An example of flare removal will now be described. First, a vertical edge 
detection is performed using a high pass filter, such as a vertical Huff filter. A second 
vertical edge detection is the performed using a medium pass filter. The data gathered 
from the two vertical edge detections are then AND together to define the area of the 
5 flare. A filter is then created to remove the area of the flare from the image. The system 
then determines the color of pixels on each side of the now blank area which used to be 
the area of the flare. A weighted average of the colors on each side of the area are used 
to fill the blank area. For example, if the pixels on one side of the blank area are red and 
the pixels are blue on the other side of the blank area, the blank area will be filled in such 

10 a manner that the color turns from red to blue as the pixels of the blank area are filled. 

In addition, preprocessing circuitry 120 may perform other types of special effects 
and image filtering techniques for each wide-angle image in each frame of wide-angle 
video. For example, color-matching, seam-blending of the two hemispheric images and 
known video filtering techniques can be performed on each wide-angle image. Also, 

15 audio tracks or audio captured simultaneously with the wide-angle video can be added to 
the wide-angle video. 

What has been described are the preferred embodiments of the present invention. 
It, however, will be apparent to those skilled in the art that it is possible to embody the 
invention in specific forms other than those disclosed in the preferred embodiments 

20 described above. This may be done without departing from the spirit of the invention, 
and the preferred embodiments are merely illustrative and should not be considered 
restrictive in any way. The scope of the invention is given by the appended claims, rather 
than the preceding description. 
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1 1. A method comprising steps of: 

2 capturing wide-angle video that includes a plurality of sequential frames, each 

3 frame having at least one wide-angle image; and 

4 replacing a portion of at least one of each wide-angle image in at least one of said 

5 frames with a replacement image. 

1 2. The method of claim 1, wherein the step of replacing a portion of at least one of each 

2 wide-angle image further includes preliminary selecting the portion of said at least 

3 one of each wide-angle image. 

4 3 . The method of claim 2, wherein said step of replacing a portion of at least one of said 

5 each wide-angle image further includes replacing the selected portion of said at least 

6 one of each wide-angle image. 

1 4. The method of claim 3, wherein the replacement image is a selected portion of a 

2 wide-angle image in a second frame in said plurality of frames. 

1 5. The method of claim 4, wherein the selected portion of the replacement image has 

2 substantially the same shape as the selected the portion for replacement of said at 

3 least one of each wide-angle image. 

1 6. The method of claim 4, wherein the selected portion of the replacement image 

2 represents an unobstructed view of the surrounding environment 

1 7. The method of claim 4, wherein the wide-angle image in the second frame is captured 

2 subsequent to said at least one of each wide-angle image having a replaced portion. 

1 8. The method of claim 1, wherein the replacement image is warped to the same 

2 curvature of the replaced portion of said at least one of each wide-angle image. 
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1 9. The method of claim 1, wherein the replacement image has substantially the same 

2 shape as the replaced portion of said at least one of each wide-angle image. 

1 10. The method of claim 1, wherein said at least one wide-angle image in each frame has 

2 a field-of-view of at least 180 degrees. 

1 11. The method of claim 10, wherein said at least one wide-angle image in each frame 

2 includes two wide-angle images. 

1 12. The method of claim 1 1, wherein the two wide-angle images are seemed together to 

2 form a spherical image. 

1 13. The method of claim 1 , further comprising steps of: 

2 dividing said at least one wide-angle image in each frame into a plurality of areas; 

3 comparing an intensity of a pixel in at least one of said areas to an adaptive 

4 threshold associated with an average intensity of the pixels in said at least one of said 

5 areas. 

1 1 4. The method of claim 1 , further including a step of: 

2 detecting flare in said at least one wide-angle image in each frame by comparing 

3 the change in intensity of each pixel in each row of said at least one wide-angle image 

4 in each frame to two threshold values; and 

5 adding pixels in each frame to remove the detected flare. 

1 15. An apparatus comprising: 

2 a wide-angle video image capture device that captures a plurality of sequential 

3 frames, each frame having at least one wide-angle image; and 

4 preprocessing circuitry that receives digital wide-angle video images of said at 

5 least one wide-angle image in each frame and replaces a selected portion of at least 

6 one of each wide-angle image in at least one of said frames with a replacement image. 
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1 16. The apparatus of claim 15, wherein the replacement image is a selected portion of a 

2 wide-angle image in a second frame in said plurality of frames. 

1 17. The apparatus of claim 16, wherein the selected portion of the replacement image has 

2 substantially the same shape as the replaced portion of said at least one of each wide- 

3 angle image in at least one frame. 

1 18. The apparatus of claim 16, wherein the selected portion of the replacement image 

2 represents an unobstructed view of the surrounding environment. 

1 19. The apparatus of claim 16, wherein the wide-angle image in the second frame is 

2 captured subsequent to said at least one of each wide-angle image in at least one 

3 frame having a replaced portion. 

1 20. The apparatus of claim 15, wherein the replacement image is a planar image warped 

2 to the same curvature of the replaced portion of said at least one of each wide-angle 

3 image in at least one frame. 

1 21. The apparatus of claim 15, wherein the replacement image has substantially the same 

2 shape as the replaced portion of said at least one of each wide-angle image in at least 

3 one frame. 

1 22. The apparatus of claim 15, wherein said at least one wide-angle image in each frame 

2 has a field-of-view of at least 1 80 degrees. 

1 23. The apparatus of claim 22, wherein said at least one wide-angle image in each frame 

2 includes two wide-angle images. 

1 24. The apparatus of claim 23, wherein the two wide-angle images are seemed together to 

2 form a spherical image. 
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25. The apparatus of claim 15, wherein said at least one wide-angle image in each frame 
is divided into a plurality of areas, and the intensity of each pixel in at least one of 
said plurality of areas is compared to an adaptive threshold. 

26. The apparatus of claim 25, wherein the intensity of a pixel in said at least one of said 
plurality of areas is adjusted, when the intensity of the pixel is not equal to the 
adaptive threshold. 

27. The apparatus of claim 25, wherein the adaptive threshold is associated with the 
average intensity of the pixels in the said at least one said plurality of areas. 

28. The apparatus of claim 15, wherein the preprocessing circuitry compares the change 
in intensity of pixels in each row of said at least one wide-angle image in each frame 
to two threshold values to detect flare, and adds pixels in each frame to remove the 
detected flare. 
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WO 00/60853 PCT/US00/09464 
METHOD AND APPARATUS FOR PROVIDING VIRTUAL PROCESSING 
EFFECTS FOR WIDE-ANGLE VIDEO IMAGES 

This application makes reference to, incorporates herein and claims all benefits 
accruing under 35 U.S.C. §1 19(e) by virtue of a provisional patent application earlier 
filed in the United States Patent and Trademark Office on April 8, 1999, entitled System 
and Method For the Capture and Display of Spherical Video Images, (Attorney Docket 
No. 1096.79941). This application also incorporates by reference the following U.S. Pat. 

Applications filed on the same date as the present application: Serial No. , (Attorney 

Docket No. 01096.86946) entitled "Remote Platform for Camera"; Serial No. , 

(Attorney Docket No. 01096.86942) entitled "Virtual Theater"; and Serial No. , 

(Attorney Docket No. 01096.84594) entitled "Immersive Video Presentations". Also, 
U.S. Pat Nos. 5,990,941 and 6,002,430 are incorporated by reference. 

TECHNICAL FIELD OF THE INVENTION 

The present invention generally relates to processing effects for wide-angle 
images. More particularly, the present invention relates to special effects processing that 
provides image portion replacement for wide-angle video images, shifted image portion 
replacement for wide-angle video images and other special effects for wide-angle video. 

BACKGROUND OF THE INVENTION 

A goal of imaging technology is to make the observer feel as though he or she is 
part of an image. U.S Pat Nos. 5,990,941 and U.S. Pat. No. 6,002,430, currently assigned 
to the assignee of the present application, describe wide-angle image viewing systems 
and methods. The patents provide full freedom viewing inside of a complete spherical 
image composed of one or two combined hemispheric images. As a result, there are no 
bounds on the user's freedom to view in any direction. 

U.S. Pat. No. 6,002,430 patent discusses techniques for capturing first and second 
images, each having a field-of-view of 180 degrees or greater. In one embodiment two 
cameras are placed in a "back-to-back" arrangement having lenses with a 180 degree or 
greater field of view (e.g., fisheye lenses). An imaging element or elements capture the 
images from each lens. The captured images are seamed together to form a single 

i 
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spherical image. Also, U.S. Pat. No. 6,002,430 patent includes a viewing engine that 
provides a real-time interactive image window that can be directed anywhere in the 
spherical image. Preferably, a personal computer system runs perspective correction 
algorithms to create the image window. Other wide-angle viewing systems are known, 
but known systems generally do not provide special effects for wide-angle video images. 

Reber et al, in U.S. Pat. No. 5,706,421, discloses placing an animated character in 
a wide-angle image. An animated character generator superimposes the animated 
character onto a perspective corrected image in a spherical wide-angle image. However, 
the animated character is over-laid on the spherical image, and the Reber et al. does not 
disclose performing special effects on non-perspective corrected images in wide-angle 
video images. 

Special effects can be used to enhance the quality of video images and a need 
exists for providing special effects for wide-angle video images. 

SUMMARY OF THE INVENTION 

An aspect of the present invention is to provide an apparatus and methods that 
provides special effects for wide-angle video images. 

In accordance with the principles of the present invention, a method is provided 
for capturing wide-angle video that includes a plurality of sequential frames, each frame 
having at least one wide-angle image, and replacing a portion of at least one of each 
wide-angle image in at least one of the frames with a replacement image. The step of 
replacing a portion of at least one of each wide-angle image includes replacing the 
portion with a portion of a wide-angle image in a second frame captured subsequently to 
the frame having the replaced portion. The portion of the image in the second frame may 
have the same shape as the replaced portion and represents an unobstructed view of the 
surrounding environment. Alternatively, the step of replacing a portion of at least one of 
each wide-angle image includes replacing the portion with a planar image that is warped 
to the same curvature of the replaced portion and has the same shape as the replaced 
portion. 

In accordance with the principles of the present invention, an apparatus is 
provided that includes a wide-angle video image capture device that captures wide-angle 
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video in a plurality of sequential frames, each frame having at least one wide-angle 
image, and preprocessing circuitry that receives digital wide-angle video images of the at 
least one wide-angle image in each frame and replaces a selected portion of at least one 
of each wide-angle image in at least one of said frames with a replacement image. The 
5 replacement image includes a selected portion of a wide-angle image in a second frame 
captured subsequently from the frame having a replaced portion. The selected portion of 
the replacement image has substantially the same shape as the replaced portion and 
represents an unobstructed view of the surrounding environment. Alternatively, the 
replacement image is a planar image warped to the same curvature and having the same 
0 shape as the replaced portion. 

In accordance with the principles of the present invention, a method and apparatus 
is provided for performing image-intensity scaling. A wide-angle video image is divided 
into a plurality of areas. An adaptive threshold, associated with the average image 
intensity of pixels in each area, is calculated for each area. The intensity of a pixel in at 
> least one of said areas is compared to the adaptive threshold for the area. The pixel 
intensity is adjusted, when the intensity is not equal to the adaptive threshold. 

In accordance with the principles of the present invention, a method and apparatus 
is provided for detecting and removing flare. Preprocessing circuitry compares the 
change in intensity of pixels in each row of at least one wide-angle image in each frame 
to two threshold values to detect flare, and adds pixels in each frame to remove the 
detected flare. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The invention is described in detail with regard to the following figures, 
wherein like numeral references refer to like elements, and wherein: 

FIG. 1 is a schematic block diagram of a wide-angle video image system in 
accordance with embodiments of the present invention; 

FIG. 2 is a schematic block diagram of an image capturing system in accordance 
with embodiments of the present invention; 

FIG. 3 represents two spherical images combined into a single spherical image in 
accordance with embodiments of the present invention; 
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FIG. 4 represents a data structure of a video frame in accordance with embodiments 
of the present invention; 

FIG. 5 represents an image portion replacement special effect in accordance with 
embodiments of the present invention; 

FIG. 6 represents a shifted image portion replacement special effect in accordance 
with embodiments of the present invention; and 

FIG. 7 shows flare in a wide-angle video image in accordance with embodiments of 
the present invention. 

FIG. 8 illustrates a sampling and blending operation to fill a blank section of an 
image in accordance with embodiments of the present invention. 

FIG. 9 illustrates a helicopter with a camera system mounted thereon according to 
embodiments of the present invention. 

FIG. 10 illustrates creating a 3-dimensional area in which a helicopter can be 
controlled according to embodiments of the invention. 

FIG. 1 1 illustrates a seaming operation according to embodiments of the invention. 

DETAILED DESCRIPTION OF THE INVENTION 

A system for creating wide-angle video images, such as spherical video images is 
shown in FIG.l. Wide-angle video image system 100 includes video image capture 
system 110, preprocessing circuitry 120, viewing engine 130 connected to display 
monitor 140 and/or storage 150 connected to preprocessing section 120. Video images 
output from image capture system 110 are processed, possibly along with other data, in 
preprocessing circuitry 120. Preprocessing circuitry. 120, for example, adds special 
effects to the received video images. The output of pre-processing circuitry can be stored 
in storage 150 and/or processed in viewing engine 130, such as the viewing engine 
disclosed in U.S. Pat. 6,002,430, for display. 

Image capture system 1 10 captures at least one wide-angle image. For example, 
any of the video image capture systems described in co-pending U.S. Pat. application 
(Attorney Docket No. 1096.82594) may be used. The output of image capture system 
110 may be frames of film, each frame including at least one wide-angle image. The 
video images are either captured with a digital image capture device or converted to 

* »■ 
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digital images. For example, as described in co-pending U.S. Patent Application Serial 

No - (Attorney Docket No. 1096.82594) and as illustrated in FIG. 2, an image 

capture device in image capturing system 110 may include two video cameras 210 and 
220 having fisheye lenses 250 and placed in a back-to-back arrangement for 
5 simultaneously capturing two video images. Video cameras 210 and 220 can be 
mounted, for example, on a base 230 attached to robot platform 235 located on mobile 
robot 240. Other structures for supporting video image capture devices, such as a tripod, 
may be used, and other vehicles for carrying an image capture device, such as a 

helicopter (as described in U.S. Patent Application Serial No. (01096.86946) 

10 entitled "Remote Platform for Camera" filed herewith), may be used. Video camera 210 
is orientated to capture the environment towards the front of robot 240 and video camera 
220 is oriented to capture the environment towards the rear of. the robot 240. Video 
cameras, however, can be orientated in other opposing directions. For example, co- 
pending U.S. Patent Application Serial No. (Attorney Docket No. 1096.86946) 

15 describes a platform on a robot for carrying an image capture device. Video cameras 210 
and 220 capture the environment digitally or on film in two hemispheric images 310 and 
320, which are eventually seamed together to form a spherical image 330 shown in FIG. 
3B. According to one embodiment, as illustrated in Figure 1 1, the raw video images of 
the first two hemispherical images 1100 can be corrected in 1104 and then seamed 

20 together using the correction determined in 1 104 to form seamed images 1 102. The next 
N frames can then be corrected using the correction determined for the first frame in 
1106. Thus, the correction for every frame need not be determined. Rather, the 
correction factor need only be determined every 10, 20, 50, etc frames, as conditions 
dictate. 

25 Image capture system 1 10 preferably records data representing pan, tilt and rotate 

coordinates of the image capture device simultaneously with the captured images. 
Similarly, time of day and date clock data may preferably be recorded simultaneously 
with captured images. Also, audio data may be recorded simultaneously with captured 
images. Preferably, a digital or electric microphone may record audio data and be 

30 mounted and directed in the same direction as a camera lens to capture the sounds 
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coming from the direction of image capture as known in the motion picture and television 
programming arts. 

As described above, the camera system may be mounted on a vehicle such as a 
helicopter, as illustrated in Figure 9. The camera system 901 is mounted on a camera 
mount 903 which is connected to the helicopter 905. The video and audio signals 
captured by the camera system 901 are recorded in a memory 909. The recorded signals 
can then be downloaded once the helicopter returns from the trip. Alternatively, some or 
all of the data can be transmitted back to a home base from the helicopter 905 using for 
example a transceiver 911. The recorded signals may be transferred to the transceiver 
91 1 by a computer 909 connected to both the memory 907 and the transceiver 911. It 
will be understood that the memory 907 may be part of the computer 909. The recorded 
signals can be transmitted in either a corrected format or an uncorrected format, wherein 
the computer 909 performs the correction. In addition, the entire image signals can be 
transmitted using high resolution or low resolution. In addition, only part of the image 
signals need be transmitted back to the home base, for example, only the images from the 
forward looking camera or a portion of the images in the vicinity of the direction of view. 
One advantage of only transmitting a portion of the entire images is that the decrease in 
transmission time allows the vehicle to be remotely piloted over long distances. 

Likewise, global positioning system data may be captured simultaneously with 
images or by pre-recording or post-recording the location data as is known from the 
surveying art. The precise latitude and longitude global coordinates of each image as it is 
captured can be recorded. 

The global positioning system data may be used to create a 3-dimensional area in 
which a vehicle, such as a helicopter, can operate. For example, a cruise ship 1001 may 
offer passengers the ability to take a "virtual" helicopter ride to a nearby island 1007 as 
illustrated in Figure 10. By using the global positioning system data, a box 1005 within 
which a helicopter 1003 can fly can be created. The passenger can then remotely pilot 
the helicopter within the defined box. The system can be defined such that if the 
helicopter leaves the box 1005, the helicopter will immediately return back into the box 
or return to the cruise ship. 
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Preprocessing circuitry 120 performs special effects, described in the following 
sections, on the received wide-angle images. Also, preprocessing circuitry 120 includes a 
processor and memory that may be provided at the video capture system 1 10 or remotely 
at, for example, a personal computer. Preprocessing circuitry 120 can assemble received 
wide-angle video images and other recorded data, for example, in video frames having a 

data structure described in co-pending U.S. Patent Application Serial No. 

(Attorney Docket No. 01096.84594) entitled "Immersive Video Presentations" and 
shown in FIG. 4. Also, preprocessing circuitry 120 may convert wide-angle video from 
image capture device 1 10 into conventional formats, such as AVI or MPEG. 

The wide-angle video images, such as the hemispheric images, and associated 
data may be stored in storage section 150. Storage section 150 may include video tape, 
CD-ROM, semiconductor devices, magnetic or magneto-optical disks, or laser disks as 
the storage medium. Also, the wide-angle images may be displayed using viewing 
engine 130, which may include the viewing engine in U.S. Pat. No. 6,002,430 patent. 

As described above, preprocessing circuitry .120 can provide special effects for 
wide-angle images captured by video image capture system 110. In a preferred 
embodiment of the present invention, an image portion replacement special effect for 
wide-angle images is provided using preprocessing circuitry 120. 

Image portion replacement includes selecting at least one waiped wide-angle 
image from image capture system 1 10 and selecting a portion of the wide-angle image for 
replacement with a predetermined image. For example, using the arrangement described 
above and shown in FIG. 2, video cameras 210 and 220 having fisheye lenses capture 
wide-angle video images that are warped (e.g., the captured wide-angle video images are 
substantially hemispheric), as shown in FIG. 3 A. FIG. 5 illustrates image portion 
replacement for wide-angle image 320 from video camera 220. The video images are 
mapped using, for example, spherical coordinates, and a portion 510 of the images is 
selected for replacement using, for example, the spherical coordinates. Selected portion 
510 is removed and a predetermined image 520 is placed in the same location as selected 
portion 510. Predetermined image 520 can have the same shape as the selected portion 
510 and is warped to have the same curvature of the selected portion of the wide-angle 
image. 
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The warped predetermined image 520 that replaces selected portion 510 of the 
wide-angle image may be created, for example, using a conventional computer-aided 
design (CAD) program having the capability to create warped images. Alternatively, a 
planar image may be created, for example, using a well-known paint program and warped 
to have the same curvature of the wide-angle image. For example, transforms, which 
may be derived from transforms disclosed in U.S. Pat. No. 6,002,430 patent, can be used 
to convert the planar image to a warped image that has the same curvature as the selected 
portion of the wide-angle image. Also, the planar image that is warped can have the 
same size as the selected portion for replacement. 

Additionally, the image portion replacement special effect may be provided for 
video images and still images. For video images, a selected portion in one or more wide- 
angle images in one or more frames of the video images is replaced with warped 
predetermined image 520. 

In another preferred embodiment of the present invention, shifted image portion 
replacement for wide-angle video images is provided using preprocessing circuitry 120. 
Shifted image portion replacement, similar to the image portion replacement special 
effect described above, allows a selected portion of a video image to be replaced with a 
selected portion of another captured video image. For example, when wide-angle video 
images from image capture system 1 10 are stored and'or converted to a video file, which 
may include conventional formats such as AVI or MPEG, a selected portion of a wide- 
angle video image from one video frame can be replaced with an associated selected 
portion of a wide-angle video image from another video frame. 

FIGS. 6A-6B illustrates shifted image portion replacement. Using, for example, 
video image capture system 1 10 illustrated in FIG. 2, video cameras 210 and 220 capture 
the environment in the forward and aft directions of robot 240 as robot 240 moves 
through the environment. The captured video images include a series of frames that 
include frames 610-630. Each frame includes, for example, two hemispheric images 310 
and 320 captured from video cameras 310 and 320 respectively. Frames 610-630 can 
include other information (not shown) associated with each hemispheric image set, such 
as position and audio data, as described above and illustrated in FIG. 4. 
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As described above, hemispheric images 310 and 320 represent wide- 
angle video images having a field-of-view of at least 180 degrees. Therefore, a portion of 
the video images may be obstructed, for example, by a structure supporting the video 
capture device or a vehicle carrying the image capture device. For example, video 
images from video cameras 210 and 220 are obstructed by platform 235 of robot 240. 
Frames 610-630 each include front and back hemispheric images 310 and 320 
representing the surrounding environment. Platform 235, captured at the bottom of each 
image, obstructs the field-of-view for the environment below platform 235. However, 
objects in the environment obstructed by platform 235 in images 622 and 624 were not 
obstructed in image 612 in the previous frame. The obstructed areas of images 622 and 
624 are 626 and 629, respectively. The images being blocked by 626 and 629 were 
previously recorded in image 612 as images 617 and 618. In order to approximate the 
blocked images in images 622 and 624, the system replaces image area 626 with image 
area 617 and replaces image are 629 with a rotated version of image area 618. Likewise, 
the blocked areas of the image in images 626 and 629 may be obtained from image 622. 

Another example is illustrated in Figure 6B in which the camera system is 
mounted, for example, below and in front of a helicopter, wherein the forward image is 
not blocked and the backward image is partially blocked along the top of the image by 
the helicopter and/or camera mount. In this example, a forward image is used to correct a 
fill in a blocked region of a backward image of a previous frame. For example, in the 
present frame j, the forward image 660 is not blocked by the helicopter but the backward 
image 662 is partially blocked 664 by the helicopter and/or mount. The image area 668, 
which is an inverse of the blocked area 654 in image 652, may be obtained from image 
660 and rotated by 180 degrees to approximately fill the blocked image 654. Likewise, 
the image area 676 may be used to approximately fill the blocked image area 664 in 
image 662. It is appreciated that the forward looking image may need to be converted in 
dimension and shape to comport with the outline of the supporting structure. 

According to another embodiment of the invention, the portions of the images that 
are blocked by a camera mount or a transportation vehicle can also be filled by sampling 
the regions around the blocked image and creating a blending image as illustrated in 
Figures 8A-8B. In Figure 8A, the portion of the image that is blocked is represented by 
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810. A region around the blocked image can then be designated by the system as the area 
820. As illustrated in Figure 8B, samples 830, 840 and 850 can be taken from the area 
820 and blended together to fill in the region 810. An illustrative sampling and blending 
process is disclosed in PCT Application No. PCTYUS99/07667 filed on April 8, 1999, 
which is incorporated herein by reference. 

A subsequent frame that includes objects of the environment previously 
obstructed by, for example, a video image capturing device support structure, can be 
identified using the velocity of the vehicle carrying the image capture device, the shape 
of the selected portion of the video image representing the obstructed objects and the 
image capture rate of the video image capturing device. In addition the position of the 
unobstructed objects in the subsequent frame can be identified using the same 
information if the vehicle travels in one direction between the time the video image in the 
frame having the obstructed objects is captured (e.g., frame 610) and the time the video 
image in the subsequent frame having the unobstructed objects (e.g., frame 630). 

Therefore, it may be necessary to increase the image capture rate when the vehicle 
carrying the video image capture device is travelling at a high rate of velocity in order to 
minimize nonlinear movement of the vehicle between the time the video images in the 
relevant frames are captured. Alternatively, if position data for the vehicle carrying the 
video image capture device is available, such as provided by a conventional GPS 
receiver, the position of the unobstructed objects in a subsequent frame may be 
determined regardless whether the vehicle changes direction between the time the video 
images in the relevant frames are captured. 

Also, preprocessing circuitry 120 can compare the image intensity of the 
replacement selected portion, such as selected portion 650 in frame 630, with the image 
intensity of surrounding portions of the video image, such as portions of video image 
310 surrounding selected portion 650 in frame 610. If the comparison yields a contrast 
larger than a predetermined value, a predetermined image, such as predetermined image 
520 described in the previous embodiment, can be used as a replacement portion. The 
predetermined image can have the same shape and is warped to the same curvature of the 
selected portion. Alternatively, if the comparison yields a contrast larger than a 
predetermined value, the selected portion may not be replaced. Also, instead of image 
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intensity, other aspects of the selected portion or the entire image that are known in the 
art (e.g., aspects related to image quality) may be evaluated for determining whether to 
replace the selected portion of the video image. 

Additionally, other special effects can be performed using preprocessing circuitry 
120. For example, preprocessing circuitry 120 can provide image intensity scaling for 
one or more wide-angle images in each video frame that emulates the functionality of a 
mechanical iris by shifting the intensity of the displayed video image based upon, for 
example, predetermined parameters or commands entered by the user. Image intensity 
may vary throughout a wide-angle vide image. For example, the image may be bright on 
top and dark on the bottom or bright in the center and dark on the edges. Preprocessing 
circuitry 120 can change the image intensity of pixels to blend the bright and dark areas 
of a wide-angle video image. The wide-angle image can be divided into a plurality of 
areas. The average image intensity of each area is calculated. The image intensity of 
each pixel in each area is compared to, for example, the average image intensity of each 
pixel's respective area. The image intensity of each pixel may be adjusted accordingly 
(e.g., adjusted to be brighter or darker), if the pixel's image intensity is not equal to the 
threshold. The threshold can, for example, be a range of values associated with the 
average intensity of the respective area. 

Preprocessing circuitry 120 may also provide flare detection and removal. FIG. 7 
shows flare 710, comprised of a vertical line of high intensity pixels, for wide-angle 
image 720. Vertical edge detection can be applied for each wide-angle video image in 
each frame for detecting flare. Each image includes a plurality of rows of pixels. The 
difference, for example, between the average intensity of each pixel in a row is compared 
to a first threshold that, for example, allows for a relatively large change in intensity to 
detect high intensity pixels. If an entire vertical line of pixels having abnormally high 
change in intensity is not detected, another comparison can be made to a second, less 
sensitive threshold, that allows for a relatively smaller change in intensity to detect pixels 
having abnormal high change in intensities. The results of the two comparisons can be 
correlated to detect flare 710. Once flare is detected, pixels having an intensity relative 
to the intensity of pixels proximate the flare may be filled in each row across the flare. 
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An example of flare removal will now be described. First, a vertical edge 
detection is performed using a high pass filter, such as a vertical Huff filter. A second 
vertical edge detection is the performed using a medium pass filter. The data gathered 
from the two vertical edge detections are then AND together to define the area of the 
flare. A filter is then created to remove the area of the flare from the image. The system 
then determines the color of pixels on each side of the now blank area which used to be 
the area of the flare. A weighted average of the colors on each side of the area are used 
to fill the blank area. For example, if the pixels on one side of the blank area are red and 
the pixels are blue on the other side of the blank area, the blank area will be filled in such 
a manner that the color turns from red to blue as the pixels of the blank area are filled. 

In addition, preprocessing circuitry 120 may perform other types of special effects 
and image filtering techniques for each wide-angle image in each frame of wide-angle 
video. For example, color-matching, seam-blending of the two hemispheric images and 
known video filtering techniques can be performed on each wide-angle image. Also, 
audio tracks or audio captured simultaneously with the wide-angle video can be added to 
the wide-angle video. 

What has been described are the preferred embodiments of the present invention. 
It, however, will be apparent to those skilled in the art that it is possible to embody the 
invention in specific forms other than those disclosed in the preferred embodiments 
described above. This may be done without departing from the spirit of the invention, 
and the preferred embodiments are merely illustrative and should not be considered 
restrictive in any way. The scope of the invention is given by the appended claims, rather 
than the preceding description. 
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We claim: 

1 . A method comprising steps of: 

capturing wide-angle video that includes a plurality of sequential frames, each 
frame having at least one wide-angle image; and 

replacing a portion of at least one of each wide-angle image in at least one of 
said frames with a replacement image. 

2. The method of claim 1, wherein the step of replacing a portion of at least one of 
each wide-angle image further includes preliminary selecting the portion of said at 
least one of each wide-angle image. 

3. The method of claim 2, wherein said step of replacing a portion of at least one of 
said each wide-angle image further includes replacing the selected portion of said 
at least one of each wide-angle image. 

4. The method of claim 3, wherein the replacement image is a selected portion of a 
wide-angle image in a second frame in said plurality of frames. 

5. The method of claim 4, wherein the selected portion of the replacement image has 
substantially the same shape as the selected the portion for replacement of said at 
least one of each wide-angle image. 

6. The method of claim 4, wherein the selected portion of the replacement image 
represents an unobstructed view of the surrounding environment. 

7. The method of claim 4, wherein the wide-angle image in the second frame is 
captured subsequent to said at least one of each wide-angle image having a 
replaced portion. 

8. The method of claim 1, wherein the replacement image is warped to the same 
curvature of the replaced portion of said at least one of each wide-angle image. 

9. The method of claim 1, wherein the replacement image has substantially the same 
shape as the replaced portion of said at least one of each wide-angle image. 
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10. The method of claim 1, wherein said at least one wide-angle image in each frame 
has a field-of-view of at least 1 80 degrees. 

1 1. The method of claim 10, wherein said at least one wide-angle image in each frame 
includes two wide-angle images. 

5 

5 12. The method of claim 11, wherein the two wide-angle images are seemed together 
to form a spherical image. 

1 3. The method of claim 1 , further comprising steps of: 

dividing said at least one wide-angle image in each frame into a plurality of 

areas; 

10 comparing an intensity of a pixel in at least one of said areas to an adaptive 

threshold associated with an average intensity of the pixels in said at least one of 
said areas. 

14. The method of claim 1, further including a step of: 

detecting flare in said at least one wide-angle image in each frame by 
15 comparing the change in intensity of each pixel in each row of said at least one 
wide-angle image in each frame to two threshold values; and 
adding pixels in each frame to remove the detected flare. 

15. An apparatus comprising: 

a wide-angle video image capture device that captures a plurality of sequential 
20 frames, each frame having at least one wide-angle image; and 

preprocessing circuitry that receives digital wide-angle video images of said at 
least one wide-angle image in each frame and replaces a selected portion of at 
least one of each wide-angle image in at least one of said frames with a 
replacement image. 

25 

25 16. The apparatus of claim 15, wherein the replacement image is a selected portion of 
a wide-angle image in a second frame in said plurality of frames. 
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17. The apparatus of claim 16, wherein the selected portion of the replacement image 
has substantially the same shape as the replaced portion of said at least one of 
each wide-angle image in at least one frame. 

18. The apparatus of claim 16, wherein the selected portion of the replacement image 
represents an unobstructed view of the surrounding environment. 

19. The apparatus of claim 16, wherein the wide-angle image in the second frame is 
captured subsequent to said at least one of each wide-angle image in at least one 
frame having a replaced portion. 

20. The apparatus of claim 15, wherein the replacement image is a planar image 
warped to the same curvature of the replaced portion of said at least one of each 
wide-angle image in at least one frame. 

21. The apparatus of claim 15, wherein the replacement image has substantially the 
same shape as the replaced portion of said at least one of each wide-angle image 
in at least one frame. 

22. The apparatus of claim 15, wherein said at least one wide-angle image in each 
frame has a field-of-view of at least 1 80 degrees. 

23. The apparatus of claim 22, wherein said at least one wide-angle image in each 
frame includes two wide-angle images. 

24. The apparatus of claim 23, wherein the two wide-angle images are seemed 
together to form a spherical image. 

25. The apparatus of claim 15, wherein said at least one wide-angle image in each 
frame is divided into a plurality of areas, and the intensity of each pixel in at least 
one of said plurality of areas is compared to an adaptive threshold. 
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26. The apparatus of claim 25, wherein the intensity of a pixel in said at least one of 
said plurality of areas is adjusted, when the intensity of the pixel is not equal to 
the adaptive threshold. 

27. The apparatus of claim 25, wherein the adaptive threshold is associated with the 
average intensity of the pixels in the said at least one said plurality of areas. 

28. The apparatus of claim 15, wherein the preprocessing circuitry compares the 
change in intensity of pixels in each row of said at least one wide-angle image in 
each frame to two threshold values to detect flare, and adds pixels in each frame to 
remove the detected flare. 
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